Repository navigation
Keep fan-out workflow tasks linear in the member count (#1231, #1203) - #1235
Merged
Merged
Conversation
A 300-member all() of 10 KB activities took 134 s to schedule and 58-85 s to finish on 2.32.0, so the default 60-second queue job timeout killed the final workflow task on every retry. Profiling the real jobs on SQLite and MySQL found per-member steps that rescanned the whole run: - Publishing each member's task after commit rebuilt the run's whole summary, in its own autocommit transaction (90% of the scheduling task). RunWorkflowTask now publishes the tasks it committed in bounded batch transactions and refreshes each affected run summary once. - Replay checked every member with WorkflowStepHistory scans over the whole history, and the executor, ParallelChildGroup, ChildRunHistory and RunLineageView looked up each member's events, links and policy outcomes the same way. Each lookup family now indexes the loaded history once (HistoryEventIndex), keyed by the collection and invalidated when it grows. A child closure's parent projection was cubic in the width before this. - Activity starts refresh only their own timeline points, like outcomes, and the scoped timeline and failure snapshots skip decoding payloads whose stored text cannot name the activity or a failure. Recorded history and the final projection rows are identical to 2.32.0 for activity and child fan-outs. New replay-regression evidence records both fan-outs; its consumer proves lookups for one member no longer read other members' events (it fails on main) and that a live run records the evidence history unchanged. V2FanOutWidthScalingTest bounds 300 members to 2.5 times linear growth over 30, and the fan-out scaling job runs the literal 1,000-activity and 1,000-child cases. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Binding each new child to its parent's namespace read the parent run row twice per child. On PostgreSQL those reads slowed as the scheduling task kept updating that row, and they were 40% of a 300-child scheduling task (41 s -> 27 s). TaskScopeMemo remembers the value only while one queued workflow task runs, so no value is trusted across tasks. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Measured locally at 201 s on MySQL 8.4 and 297 s on PostgreSQL 17 under load. The MySQL weight adds headroom for its pcov coverage shards and hosted runners; the PostgreSQL weight adds headroom for hosted runners. Co-Authored-By: Claude Opus 5.5 <[email protected]>
The first run lost a package download to an HTTP 429 from codeload. Co-Authored-By: Claude Opus 5.5 <[email protected]>
The literal 1,000-activity run showed the final workflow task growing faster than linearly (1.5 s at 100 members, 40 s at 1,000). At 500 members 57% of it was ActivityReplayIdentity::assertCall, which scanned and decoded the whole history for every replayed activity until it found that member's ActivityScheduled event. HistoryEventIndex::find now indexes a loaded history only as far as each lookup needs: one lookup stops at its match as the old scan did, and all lookups together read each event once. The replay identity check and the first-match lookups (parallel metadata, wait kind, child scheduled and resolution events, activity completion and open events) use it, so a member task that looks up one sequence no longer decodes the whole history to build an index. Recorded history and final projection rows remain identical to main. The fan-out corpus consumer now also runs the replay identity check. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…uadratic # Conflicts: # CHANGELOG.md
…1203) Every child a workflow task starts records its child-call row after the task commits. Each row committed alone, so a 1,000-child scheduling task spent about 17 ms per child waiting on its own commit on SQLite. Inside the coalescing scope those best-effort writes now join the batched post-commit work ahead of the tasks published after them, each in its own savepoint, and still report rather than throw a failure. 1,000 children: 58 s -> 39 s. Co-Authored-By: Claude Opus 5.5 <[email protected]>
) Before every fan-out member, the scheduling task counted the run's stored pending activities or open child links twice (limit guard and warning). On MySQL those counts were the largest growing step of a 1,000-child scheduling task (13 s). Within one workflow task the first check counts the rows and each later guard adds the member scheduled since; members that close meanwhile only lower the stored count, so the value is an upper bound. Below the limit (or warning threshold) the bound decides exactly like the stored count; at or above it the rows are counted again and that count is used. A new case fails a five-member fan-out at a pending limit of three with the same kind, value and configured limit on main and on this branch. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Starting 1,000 children with 10 KB arguments is the slowest workflow task: about 33 s on MySQL and 39 s on SQLite locally after this branch's fixes, against more than 85 s on hosted runners before them. Keep the bound inside the 60-second default job timeout with room for hosted-runner variance. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Hosted runners measured the literal fan-outs on this branch: 1,000 activities schedule in 11-15 s and finish in 11-14 s, but starting 1,000 children takes 86 s on SQLite and 64 s on MySQL. Run activities and children as separate jobs, and let the child job report without failing the build until #1238 brings it under the bound. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…uadratic # Conflicts: # src/V2/Support/StructuralLimits.php
Run as its own job on the branch merged with current main, hosted runners start 1,000 children in 33.3 s on MySQL and 23.6 s on SQLite, inside the 50-second bound (they took 64 s and 86 s when the child case ran after the activity case in the same process). The child jobs no longer need continue-on-error (#1238). Co-Authored-By: Claude Opus 5.5 <[email protected]>
rmcdaniel
force-pushed
the
general/1231-fanout-quadratic
branch
from
November 19, 2026 15:24
01978f3 to
d118e4e
Compare
The same code started 1,000 children in 33.3 s and then in 65.1 s on hosted MySQL runners, so the child case is around the 50-second bound on hosted hardware. Report it without failing the build until #1238 lands. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…review) Review of the coalesced publish found three faults: - A failed push recorded last_dispatch_error and repair_available_at inside its savepoint and then rethrew, so the savepoint rolled the record back. Publishing now runs in two phases. Inside the batch transaction each task gets the metadata a successful dispatch leaves, or the failure a refused dispatch records, and nothing is thrown. After the batch commits each job is pushed. A failed push records its failure on its own and restores the dispatch marks, leaving the task as an unbatched failed publish does. - A deadlock, lock wait or lost connection ends the whole transaction, but the batch caught it and kept going. Such failures (TransactionFailure) now abort the batch, including from best-effort child-call writes. The whole batch then runs once more in a fresh transaction, and a second such failure is thrown with nothing committed or pushed. - Jobs were pushed while their batch was uncommitted, so a child could finish before its call row existed. Pushing after commit closes that gap. V2CoalescedTaskPublishTest covers each fault on MySQL and SQLite; three of its four cases fail on the previous head. The fan-out corpus consumer now compares payloads independently of key order, which MySQL's JSON column changes. Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Fixes the quadratic fan-out workflow tasks in #1231 and #1203. The stuck-run part of #1231 is a proposal below and is not changed here.
This branch has current
mainmerged in (2.33.0, including #1234'sParallelGroupAdmission). Admission counts the pending members once per group before any write. The per-member guards it keeps as a backstop use this PR's running bound, so the two compose. The only conflict wasStructuralLimits.php: it keeps main's publicpendingActivityCount/pendingChildCount/guard*Countand routes the per-member guards through the bound.Profile before (2.32.0
main, real queue jobs, one task at a time)Excimer sampling plus per-query logging on SQLite and MySQL 8.4 found these per-member steps that rescan the run:
TaskDispatcher::markDispatched→RunSummaryProjector::project), each in its own autocommit transactionWorkflowStepHistory::assertCompatible/assertParallelGroupCompatiblescanned the whole history for every memberWorkflowExecutor::activityCompletionEvent/activityOpenEvent,ParallelChildGroup::metadataPathForSequence/waitKindForSequence,ChildRunHistory::*ForSequence/latestLinkForSequence,RunLineageView::latestParentClosePolicyEventForChildscanned history or links per memberActivityReplayIdentity::assertCallscanned and decoded history until each replayed activity'sActivityScheduledeventStructuralLimitspending-child and pending-activity counts ran twice per member over a growing setChildWorkflowNamespaceProjectionread the parent run row twice per childFixes
RunWorkflowTaskpublishes the tasks it committed in bounded batch transactions (50 per batch, one savepoint per task so one failure keeps the rest) and refreshes each affected run summary once. Atomic poll repair keeps its refresh inside its own transaction; faked runs keep the old path.HistoryEventIndexmemoizes per-sequence indexes over one loaded history collection, invalidated when the collection grows or is replaced. All the lookups above use it. For lookups that used to stop at the first match,find()indexes only as far as each lookup needs, so a single lookup still stops early and all lookups together read each event once.mainand on this branch.RunSummaryProjector's existing scoped-timeline condition). The scoped timeline andFailureSnapshotsskip decoding payloads whose stored JSON cannot name the activity or a failure (StoredHistoryPayload).TaskScopeMemoremembers the parent's namespace while one workflow task runs.No limits or semantics change. Recorded history, and the final summary, wait, timeline, timer and lineage rows, are identical to
mainfor activity and child fan-outs (compared field by field, masking generated ids and timestamps).Numbers
Seconds per workflow task. Each cell is scheduling / final. These are local runs on lab4 (PHP 8.3, capped containers), with member tasks completed between them. The 30/300 rows use 1 KB arguments and the 1,000 rows use 10 KB. "Before" at 30/300 is current
main(2.33.0). "Before" at 1,000 is 2.32.0, since a 1,000-member run onmaintakes hours.The 1,000-member "after" scheduling times were re-measured on the merged branch. The 1,000-member final times are from the branch before the merge (the merge does not touch the final task).
Hosted runners (Fan-out scaling jobs on this head, 1,000 members, 10 KB,
mainmerged in): activities 11.2 / 14.4 s on SQLite and 12.6 / 9.9 s on MySQL; children 23.6 / 7.3 s and then 46.6 / 13.1 s on SQLite, and 33.3 / 10.9 s, 65.1 / 16.0 s and 44.6 / 13.5 s on MySQL across runs of the same code (head a29bf78: all four jobs pass). Activities are well under the 50 s bound. Children on hosted MySQL straddle it, so the child jobs stay non-blocking (#1238).PostgreSQL 17, 300 children, scheduling: 41 s before the namespace memo and 27 s after it (1.8 s at 30). Before this change, completing 100 children did not finish within 10 minutes, because every child closure's parent projection was cubic. Now it takes 30 s.
Review fixes (head after review)
last_dispatch_errorand an immediaterepair_available_aton its own and restores the dispatch marks. The task is left as an unbatched failed publish leaves it.TransactionFailure) aborts the batch, including from best-effort child-call writes. The whole batch then runs once more in a fresh transaction. A second such failure is thrown with nothing committed or pushed.V2CoalescedTaskPublishTestcovers each fault on MySQL and SQLite. Three of its four cases fail on the previous head.Tests and evidence
V2FanOutWidthScalingTest(feature, weighted): 30 vs 300 activities and children. It fails if either workflow task grows more than 25× (2.5× linear). Onmainthe ratio is 60× (scheduling) and 65× (final) for activities.tests/Performance/FanOutWidthLiteralTest.phpand the new Fan-out scaling workflow: the literal 1,000-activity and 1,000-child fan-outs with 10 KB arguments, as separate jobs on SQLite and MySQL. It runs nightly, on dispatch and on PRs touchingsrc/V2. Each workflow task must finish under 50 s and grow at most 25× from 100 members. The activity jobs are blocking; the child jobs report withcontinue-on-erroruntil Starting 1,000 children in one workflow task still takes 64–86 s on hosted runners #1238.fan-out-activity-all-history-indexandfan-out-child-all-history-index(recorded on this branch; history identical tomain). The cold runner replays both to completion. The new consumer inV2EmbeddedReplayRegressionCorpusTestproves that the later members' lookups together read fewer than one pass over other members' events (8 here; 114 and 92 onmain), and that a live run on this code records the evidence history unchanged.The probes skip each member task's own run-summary refresh during the member phase (
MemberPhaseProjectionRole) and refresh every summary once before the measured final task, so the tests stay bounded. See the remaining work below.Remaining work (not in this PR)
Stuck run (#1231): proposal only
RunWorkflowTaskhas$tries = 1, no$timeoutand nofailed()hook. The claim commitsleasedandattempt_count + 1with a 300 s lease before execution. Underqueue:work --timeout=60with pcntl, the alarm handler fails the job and exits the worker. The execution transaction never commits. The task staysleaseduntil its lease expires, and thenTaskWatchdogredispatches it. Every retry repeats the same work and is killed again.attempt_countgrows without bound and no failure is recorded.Proposal:
workflow_task_repeatedly_lost, with repair attention and a lifecycle event. Failing the run with a typedworkflow_task_timeoutfailure would change semantics, so it should be opt-in.--timeoutis belowworkflow_task_lease_seconds, recommending--timeout=0or a value above the largest workflow task. Document this beside the README'squeue:work.Overlap
mainis merged in.RunSummaryProjector.phphas a one-line change in the existing scoped-timeline condition, outside #1229's hunks.StructuralLimits.phpwas resolved as described above.php.ymlandvalidate-regression-corpus.pyare untouched; the slow job is a separate workflow file.🤖 Generated with Claude Code